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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be tinnely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS fronn the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent tennn adjustment. See 37 CFR 1 .704(b). 

Status 

1 )S Responsive to connnnunication(s) filed on 10 May 2004 . 
2a)IEI This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1-37 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) 1EI Claim(s) 1-37 is/are rejected. h 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)0 The drawing(s) filed on is/are: 3)0 accepted or bO objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
11 )□ The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 0 Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1) Q Notice of References Cited (PTO-892) 4) □ Interview Sunnmary (PTO-413) Paper No(s). . 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) □ Infomiation Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 

1 . This action is in response to the amendment filed 5/10/2004. 

2. Claims 1-37 have been examined and are pending in the application. 



Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-15 and 19-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Hashimoto U.S Patent No. 6,119,147. 

As to claim 1, Hashimoto teaches a system comprising a multi-modal 
application {1 A, Fig. 56) comprising at least a first and a second mode process (inputs 
and outputs from the application programs, lines 23-25 column 41; speech synthesis 
unit operated as an independent process, lines 37-38 column 42) that enables user 
interaction with the application in a first modality and second modality (the system 
allows the user to interact with the system via text data or speech data, lines 50-63 
column 49; Figs. 17, 66, and 69-72); a multi-modal shell (11, Fig. 56) for managing and 
synchronizing information exchanges between the processes (controls by exchanging 
the messages transmitted from the speech unit to the application program, lines 38-41 
column 10) to enable a synchronized multi modal interaction with the application (the 
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system allows the user to interact with the system via text data or speech data, lines 50- 
63 column 49; Figs. 17, 66, and 69-72); the processes register their respective active 
commands and corresponding actions (register the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 1 0 to line 1 0 
column 1 1 ) with the multi model shell (1 1 , Fig. 56). Hashimoto does not explitcitly teach 
an API. However, Hashimoto teaches that each application program Includes a 
message I/O unit (line 9-29 column 12) wherein all of the application program 
interactions with the speech recognition system are handled by this message I/O unit. 
Therefore one of ordinary skill in the art would conclude that this message I/O unit could 
be used as an API since it provides the interface for each application program to 
interact with the speech recognition system. 

As to claim 2, Hashimoto as modified further teaches a registry having a 
registration table (program management table, line 58 column 10), managed by the 
multi-modal shell (1 1 , Fig. 56), that comprises a list of each of the registered commands 
and corresponding synchronized actions (the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 1 1) that results in both the first and second mode processes upon execution of 
a registered command (once the speech input "Finish" is transmitted, both of the 
application program can be finished by this single speech input, lines 5-8 column 27) by 
one of the first and second mode processes (inputs and outputs from the application 
programs, lines 23-25 column 41). 
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As to claim 3, Hashimoto as modified further teaches the multi-modal 
application comprises a first mono-mode application (speech synthesis unit, line 37 
column 42) for the first mode process and a second mono-mode application (application 
program, lines 41-42 column 42) for the second mode process, wherein the nnulti-modal 
shell (11, Fig. 56) manages and synchronizes information exchanges (message 
exchanges using the process communication, lines 39-40 column 42) between the first 
(speech synthesis unit, line 37 column 42) and second mono-mode applications 
(application program, lines 41-42 column 42). 

As to claim 4, Hashimoto as modified does not teach devices having user 
interface modalities. However, Hashimoto teaches (lines 51-55 column 12) the 
message system can be implemented as a server and clients system wherein the 
speech recognition unit can act as a server and the applications programs are clients. 
Therefore one of ordinary skill in the art would conclude that the clients are the devices 
with modalities wherein these devices can register their commands and corresponding 
actions with the server. 

As to claim 5. Hashimoto as modified further teaches the devices multi-modal 
shell are distributed over a network (server and client system, lines 51-55 column 12), 
and wherein the API is implemented using distributed APIs or protocols (byte stream 
type protocol, lines 54-55 column 12). 

As to claim 6, Hashimoto as modified further teaches a mechanism for 
converting a mono-mode application to a multi modal application (application program 5 
is interact with both speech input and keyboard input, Fig. 17). 
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As to claim 7, Hashimoto as modified further teaches the mono-mode 
application is a GUI application (user interface of Fig. 18), the mechanism (1 1 . Fig. 56) 
provides speech enablement (controls by exchanging the messages transmitted from 
the speech unit to the application program, lines 38-41 column 10) of the GUI 
application (user interface of Fig. 18) by registering the active commands of the GUI 
application and building a grammar for the registered commands to support the 
commands in a speech modality (register the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 11). 

As to claim 8, Hashimoto as modified further teaches a mechanism for building 
a multi-modal application (application program 5 is interact with both speech input and 
keyboard input, Fig. 17). 

As to claim 9, Hashimoto as modified further teaches the mechanism (1 1 , Fig. 
56) is used for directly programming the registry by building a registration table 
(program management table, line 58 column 10) having user-defined commands and 
corresponding actions (the recognition vocabularies and the appropriate actions that 
response to these vocabularies, line 48 column 1 0 to line 1 0 column 1 1 ) for each of the 
modalities of the multi-modal application (1A, Fig. 56). 

As to claim 10, Hashimoto as modified further teaches that the system is 
implemented on personal computers, workstations... (lines 8-12 column 1). Hashimoto 
does not explicitly disclose an operating system. "Official Notice" is taken that both the 
concept and advantage of providing for an operating system is well known and 
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expected in the art. It would have been obvious to include an operating system into the 
system of Hashimoto because it would provide the execution space for the system. 

As to claim 11, Hashimoto as modified further teaches the system is distributed 
over a network (server and client system, lines 51-55 column 12). 

As to claim 12, Hashimoto as modified further teaches the multi-modal 
application (1 A, Fig. 56) is a multi-modal browser (a mail browser, Fig, 27), comprising 
first and second browser applications (multiple application programs 2, Fig. 7). 

As to claim 13, Hashimoto as modified further teaches the first browser Is GUI 
(interface of Fig. 27) and the second browser is speech (the user can command to open 
the mail by saying "y^s", lines 54-60 column 25). 

As to claim 14, Hashimoto as modified further teaches the multi-modal shell (11, 
Fig. 56) processes a CML file (recognition result, line 16 column 25) to send modality 
specific presentation information (speech result from the speech recognition system, 
lines 14-17 column 25) in the CML file (recognition result, line 16 column 25) to the 
respective browsers (a mail browser, Fig. 27). 

As to claim 15, Hashimoto as modified further teaches the CML file (recognition 
result, line 16 column 25) encapsulates the modality-specific presentation information 
(speech result from the speech recognition system, lines 14-17 column 25) in a single 
modality-independent representation (input speech, line 12 column 25). 

As to claim 19, it is a method claim of claim 1 . Therefore, it is rejected for the 
same reasons as claim 1 above. Hashimoto as modified further teaches receiving a 
command (speech input, line 14 column 41) in a first modality (speech recognition 
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interface system, line 13 column 41); triggering an action (speech output, line 19 column 
41) in the first modality (speech recognition Interface system, line 13 column 41) and a 
corresponding action (the mail is opened when the user saying "yes", lines 54-60 
column 25) in at least a second modality (application program, lines 41-42 column 42) 
based on the received command; updating application state associated with the second 
modality (update the program management data in the program management according 
to an internal state of each application program, lines 35-39 column 76). Hashimoto 
does not explltcitly teach updating application state associated with the first modality. 
However, Hashimoto teaches (lines 37-46 column 14) the speech recognition system 
also has its own internal state; and when there is a change in this internal state, the 
application program will get a message notifies about this change. Therefore one of 
ordinary skill in the art would conclude that in a certain time, the internal state of the 
speech recognition system is updated; therefore produces the state change. 

As to claim 20, Hashimoto as modified further teaches registering active 
commands associated with the modalities (the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 1 1 ); associating, with each registered command of the modalities, an action on 
one modality and a corresponding action on the other modality (once the speech input 
"Finish" is transmitted, both of the application program can be finished by this single 
speech input, lines 5-8 column 27). 

As to claim 21, Hashimoto as modified further teaches building a command to 
action registration table (program management table, line 58 column 10) based on the 



Application/Control Number: 09/507,526 Page 8 

Art Unit: 2126 

registered commands and actions (the recognition vocabularies and the appropriate 
actions that response to these vocabularies, line 48 column 10 to line 10 column 11). 

As to claim 22, Hashimoto as modified further teaches the registration table 
(program management table, line 58 column 10) is built by a multi-modal shell (1 1 , Fig. 
56). Hashimoto does not explitcitly teach an API. However, Hashimoto teaches that 
each application program includes a message I/O unit (line 9-29 column 12) wherein all 
of the application program interactions with the speech recognition system are handled 
by this message I/O unit. Therefore one of ordinary skill in the art would conclude that 
this message I/O unit could be used as an API since it provides the interface for each 
application program to interact with the speech recognition system. 

As to claim 23, Hashimoto as modified further teaches looking up the received 
command (the recognition vocabulary lists, lines 59-60 column 10) in the registration 
table (program management table, line 58 column 10); and executing the actions 
associated with the received command on the first and second modalities (once the 
speech input "Finish" is transmitted, both of the application program can be finished by 
this single speech input, lines 5-8 column 27). 

As to claim 24, Hashimoto as modified further teaches registering a callback 
handle for each of the registered commands to notify the first and second modalities of 
completion of the actions corresponding to the registered commands (after the 
dictionary production is completed, notifies this act to the data acquisition unit 8 by a 
message indicating the completion of the dictionary production, lines 3-5 column 33). 
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As to claim 25, Hashimoto as modified further teaches executing the callback 
handle associated with the received command to trigger callback actions on the 
modalities (after the dictionary production is completed, notifies this act to the data 
acquisition unit 8 by a message indicating the completion of the dictionary production, 
lines 3-5 column 33). 

As to claim 26, Hashimoto as modified further teaches executing first thread 
associated with the received command ("Finish", line 6 column 27); triggering a 
corresponding second thread to initiate the corresponding action on the second 
modality (once the speech input "Finish" is transmitted, both of the application program 
can be finished by this single speech input, lines 5-8 column 27), 

As to claim 27, Hashimoto as modified further teaches the threads are applets 
("Yes" and "No" icons in Fig. 27). 

As to claim 28, Hashimoto as modified further teaches the threads communicate 
via socket connections (server and client system (lines 51-55 column 12) with a mail 
browser, Fig. 27)). 

As to claims 29-37, they are device claims of claims 19-21 and 23-28. 
respectively. Therefore, they are rejected for the same reasons as claims 19-21 and 
23-28 above. 

4. Claims 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hashimoto in view of Toomey U.S Patent No. 6,1 19,147. 
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As to claim 16, Hashimoto as modified does not explitcitly teach the file 
comprises a combination of declarative markup languages. Toomey teaches a multi- 
model document with a combination of declarative markup languages (text discussion, 
audio commands, graphics, and documents, lines 60-61 column 1 ). It would have been 
obvious to apply the teachings of Toomey to the system of Hashimoto because this 
document will provide the user with the convenience of interacting with the system using 
the choice of texts or speech commands. 

As to claim 17, Toomey further discloses a single file combining the declarative 
markup languages and synchronization elements to provide tight synchronization 
between the declarative markup languages (interactions are inserted into the multi- 
modal document at a point that is chronological in the meeting to create a synchronous 
meeting, lines 11-13 column 15). 

As to claim 18, Toomey further discloses a separate file for each of the 
declarative markup languages (multiple tracks in the multi-modal document, lines 59-60 
column 1). 

Response to Arguments 

5. Applicant's arguments filed have been fully considered but they are not 
persuasive. 

Applicant argued that Hashimoto reference does not teach first mode process 
that enables user interaction with the application in first modality and a second mode 
process that enables user interaction with the application in a second modality 
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(Remarks, last complete paragraph page 8 to first Incomplete paragraph page 9). In 
response, the applicant argued new limitation that was not claimed before. However, 
this limitation is still met by the cited references as disclosed in the claim rejections 
above. 

Applicant argued that Hashimoto reference does not teach managing and 
synchronizing information exchanges between processes to enable a synchronized 
multi-modal interaction with the application (Remarks, last complete paragraph page 9). 
In response, Hashimoto teaches the system allows the user to interact with the system 
via text data or speech data (lines 50-63 column 49). The reference meets the limitation 
as claimed. 

Applicant argued that Hashimoto reference does not teach updating application 
states (Remarks, last incomplete paragraph page 9 to first incomplete paragraph page 
10). In response, Hashimoto teaches updating the program management data in the 
program management according to an internal state of each application program (lines 
35-39 column 76). The reference meets the limitation as claimed. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
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mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 

examiner should be directed to The Thanh Ho whose telephone number is (571 ) 272- 

3762. A voice mail service is also available for this number. The examiner can 

normally be reached on Monday - Friday, 8:30 am - 5:00 pm. 

Any inquiry of a general nature or relating to the status of this application or 

proceeding should be directed to the receptionist whose telephone number is 703-305- 

3900. 

Any response to this action should be mailed to: 

Commissioner for Patents 

P.O Box 1450 

Alexandria, VA 22313-1450 
Or fax to: 

• AFTER-FINAL faxes must be signed and sent to (703) 872 - 9306, 

• OFFICAL faxes must be signed and sent to (703) 872 - 9306. 

• NON OFFICAL faxes should not be signed, please send to (571 ) 273 - 3762 
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